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ABSTRACT 

The National Assessment of Educational Progress was 
used as the primary model in a multistate workshop on the assessment 
of educational programs. One day each was devoted to the development 
of objectives, exercise development, sampling, administration and 
scoring, data analysis, and reporting and dissemination. This 
document concerns itself primarily with the organization and the 
evaluation of the workshop. The appendix provides an outline of the 
Colorado Assessment Program and educational assessment plans for nine 
other States* The results of a 6-month followup evaluation to 
determine the workshop's influence on State assessment plans are 
included. (Author/DN) 
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ECS Assessment Workshop 



Abstract 



A two-week assessment workshop was held in Boulder, Colorado 
on July 12-23, 1971 for per sonnel from state departments of education. 
Seventeen participants from ten states were in attendance. 

The National Assessment of Educational Progress was used as the 
primary assessment model. One day each was devoted to development 
of objectives, exercise development, sampling, administration and 
scoring, data analysis, and reporting and dissemination. In addition, 
three one-half day sessions each were devoted to presentation of three 
existing state assessment plans -- Colorado, Pennsylvania, and New York. 

Conference participants prepared an outlined plan for assessment in 
their own states, utilizing whatever model or variations seemed best to 
them. 

Conference participants completed an evaluation. All of them recom- 
mended a similar workshop for other state department personnel. All 
of them indicated that they had learned something of potential utility for 
their own state. Generally, they prefer red the more specific presenta- 
tions. All of them indicated a need for follow-up assistance as they sought 
to implement assessments in their own states. 
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Introduction 



The early 1970 f s may well be remembered in education as the 
"age of accountability. " Assessment is a term commonly associated 
with accountability, a term that suggests a method for seeking evidence 
as to whether schools (or the total educational process) are being 
accountable, are doing what they say they are doing. Many assess- 
ment efforts are in the planning stages; only a few have been imple- 
mented and have produced results. National Assessment is easily the 
most visible assessment effort in the country, and, in the opinion of 
many, the most carefully developed one. In spite of considerable initial 
hostility, National Assessment now is accepted and approved by most of 
the educational community. 

National Assessment is designed to provide national results, 
regional results, and size of community r esults . It is not designed to 
provide state or school district results. Since it was designed for effi- 
cient information-gathering in large units, it does have potential specific 
utility for states (and possibly a few very large cities). It has general 
utility as a "model 1 - assessment for almost any sized educational unit. 
The potential utility of National Assessment as a model is above-and- 
beyond its major thrust of gathering direct evidence on the achievements 
of young people; it is a side benefit that needs to be exploited. 

Already the educational research community is well aware of 
National Assessment and is highly approving. National Assessment is 
appearing as a major topic in test and measurement textbooks and is 
being taught in college classes. 

Some states have established state assessments and many others 
are planning assessments. Only a few have expressed no interest. 

The utility of National Assessment as a "field 11 model for state 
level assessments is apparent to many. The Education Commission of 
the States (ECS) has received many inquiries about National Assessment 
as a model, about National Assessment materials, etc. FCS held one 
short state assessment meeting a year and a half ago and over 20 states 
were represented. 

In an effort to make National Assessment more accessible to the 
states, and. in an effort to present alternative assessment models to the 
states a two-week assessment workshop for state department personnel was 
developed by ECS, funded by NCERD, and organized by Frank B. Womer 
and Irvin J. Lehmann in Boulder, Colorado from July 12 - 23, 1971. 
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Method 






The workshop was organized as follows 


: 




July 12 


ECS Welcome 

Assessment 

Goals and Objectives 


Wendell Pierce 
James Hazlett 
Vincent Campbell 




July 13 


Exercise Development 


Carmen Finley 




July 14 


Colorado Assessment 
Sampling 


Arthur Olson 
David Bayless 




July 15 


Sampling 
Assessment 


David Bayless 

Frank Womer and 
Irvin Lehmann 




July 16 


Pennsylvania Assessment 
New York Assessment 


Paul Campbell 
Lome Woollatt 




July 19 


Administration and Scoring 


George Johnson, Larry 
Conaway, and David Wright 




July 20 


Data Analysis 


Dale Foreman 




July 21 


Reporting and Dissemi- 
nation 


Eleanor Norris 




July 22 


Presentations of ten state 
plans 

Workshop Dinner 


Conference participants 

J. Stanley Ahmann, speaker 




July 23 


Evaluation 


Frank Womer and 
Irvin Lehmann 




Each day's activities were a combination of presentations and dis- 
cussions. The volume of questions from participants made it very 
easy to zero in on their interests. 




Representatives from each state represented in the workshop pre- 
pared an outline of an assessment plan for their own strte. They are 
presented in Appendix A. 
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Organization 



All conference arrangements were made by Mr, George Goulette 
of the University of Colorado Conference Office. Workshop sessions 
were held in three rooms of the University of Colorado Engineering 
Center. Workshop participants had a choice of housing in a University 
dormitory or a nearby motel. The University of Colorado Conference 
Office handled registration, provided audio- visual equipment, handled 
telephone messages, provided coffee breaks, provided a typist and re- 
production services, and arranged for a workshop dinner. All arrange- 
ments were handled very smoothly. 
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Staff and Consultants 



Staff: 

Womer, Frank B. 
Professor, School of Education 
The University of Michigan 
(formerly Staff Director of NAEP) 



Consultants : 

Ahmann, J. Stanley 
Staff Director 
NAEP 



Bayless, David L. 

Head of Sample Design and 

Evaluation Section 
Research Triangle Institute 

Campbel 1 , Vincent 
Associate Director, Social and 
Education Research Program 
American Institutes for Research 

Campbell, Paul 

Director, Department of Re- 
search and Statistics 

Pennsylvania Department of 
Education 

Co naway, Lar r y 

Assistant to the Director of 

Operations 
NAEP 



Lehmann, IrvinJ. , Professo 
Office of Evaluation Services 
Michigan State University 
(formerly Director of Resear 
for NAEP) 



Finley, Carmen J. 
Associate Staff Director 
NAEP 



Foreman, Dale 
Assistant to the Director/ 

Research and Analysis 
NAEP 

Hazlett, James A. 
Administrative Director 
NAEP 



Johnson, George H. 
Director of Operations 
NAEP 



Nor r is , Eleanor 

Social and Educational Re- 
search Program 

American Institutes for 
Research 



7 

i 



Consultants : (continued) 



Olson, Arthur 

Director, Assessment and Evalua- 
tion 

Colorado State Department of 
Education 



Woollatt, Lome 
Associate Commissioner 

of Education 
New York State Department 

of Education 



Pierce, Wendell H. Wright, David 

Executive Director Assistant to the Director/ 

Education Commis sion of the States Research and Analysis 

NAEP 
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Workshop Participants 



Colorado 

Haberbosch, John 
Department of Education 

Helper, John 
Department of Education 

Idaho 

Carpenter, Donald J. 

State Department of Education 

Schwartz, John A. 

State Department of Education 

Iowa 

Lidstrom, David C. 

Iowa Department of Public Instruction 
Morrison, Max 

Iowa Department of Public Instruction 
Maine 

Natale, Anthony J. 

State Department of Education 

Maryland 

Fisher, James 
Department of Education 
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Participants : (continued) 



Massachusetts 

Kraskouskas, John C. 

Massachusetts Department of Education 



Berger, Gerard 

Oregon Board of Education 

Texas 

Balboa, Arnulfo 

Texas Education Agency 

Fischer, William 
Texas Education Agency 

Hardebeck, Rich 

Texas Education Agency 

Russell, Charles 

Texas Education Agency 

West Virginia 

Berty, Ernest 
Department of Education 

Thornton, Philip F. 
Department of Education 

Wisconsin 

Daeschner , Stephen 

Wisconsin Department of Public Instruction 



Oregon 




Evaluation 



On the final day of the workshop, the participants were asked to 
complete an evaluation form (Appendix B). The purpose of this eval- 
uation was two-fold: (1) to provide the workshop staff with some 
objective (in addition to subjective) evidence concerning the impact 
of the workshop experience on the participants, and (2) to provide the 
workshop staff and Office of Education with feedback that would or 
could be of value for future workshops of this nature. 

The major questions asked in the evaluation form were as 
follows : 

1. The value of the workshop experience. 

2. The relative value of the nine major presentation/lecture/ 
discussion sessions. 

3. Recommended improvements for future workshops. 

4. The most and least valuable workshop experience. 

5. General comments. 



General Impressions of Experiences. 

By and large, the participants felt that the majority of the experi- 
ences provided them in the workshop were of vs.lue to them. The staff 
are most pleased with this inasmuch as there was a diversity of personnel 
attending and there was an attempt to provide a workshop that would be of 
benefit to all the participants rather than just to a subset of them. As 
evident from the written comments as well as informal conversations, 
the participants were pleased with the overall workshop structure, the 
course content, and the speakers. They felt that they obtained a much 
better understanding of the National Assessment plan as a result of 
attending the workshop. More important, they obtained a clearer per- 
spective of how the National Assessment plan could be modified to fit 
their own specific needs. This was very evident from the kinds of 
questions raised in the discussion. The occasional instance where a "poor 11 
rating was given such as n the speakers should hand out their prepared 
text, " n need one set of materials for every participant" can easily be 
rectified for future workshops. 
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Initially, the workshop directors had planned for more individual 
work sessions especially in the areas of developing objectives and 
writing exercises. However, there was so much discus sion (beyond 
that anticipated by the directors) that the amount of time available 
for individual work was rather limited. This no doubt accounts for, 
in large part at least, the "poor 11 rating given to independent work 
sessions. The data pertaining to this question are presented below. * 







Value 




Area 


Good 


Average 


Poor 


Overall workshop structure 


7 


7 


2 


Workshop content 


12 


4 


0 


Speakers 


11 


5 


0 


Training methods 


2 


13 


1 


Mater ials 


8 


6 


2 


Hearing other participants' 
plans 


3 


5 


2 


Independent work sessions 


0 


8 


6 


Living facilities 


8 


6 


0 



The negative comments pertaining to the overall workshop structure 
were that an attempt should have been made to present the material in 
one week rather than two by lengthening the workshop day. This same 
point was also made in the free-i-esponse section where the participants 
were asked to suggest possible improvements for future workshops of 
this nature. 



* One participant left prior to the evaluation form being handed out. 
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Relative Value of Major Presentation Sessions 



The participants were asked to rank, in order of importance, the 
nine major presentation sessions in relation to their present and 
(possible) future work in state and local assessments. Using a weight- 
ing scheme where 9 = the activity ranked first and 1 = the activity- 
ranked ninth, the mean ratings in descending order of importance 



were as follows : 

Session Mean Rating 

Sampling 7. 1 

Exercise development 6.6 

Developing objectives 6.5 

Data analysis 6.3 

Administration and scoring 5.3 

Reporting and dissemination 5.0 

Pennsylvania Plan 4.5 

Colorado Plan 2. 9 

New York Plan 1.7 



It would appear that the more technical and less familiar mater- 
ial such as sampling and exercise development were perceived by 
the participants to be of more importance than some of the State 
Assessment plans which were more general in nature and which may 
have been more familiar to them. It also was evident from the amount 
of discussion generated that the material on sampling, exercise de- 
velopment, objectives, and data analysis were very well received. Al- 
though there was discussion on the State Assessment plans presented, 
it was not as detailed and probing. The general comments noted on 
the evaluation form as well as those gathered by the staff during the 
two-week session supported the results of the evaluation form. 
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It should be kept in mind that a ranking forces some low scores 
as well as some high scores. In actuality the general rating for all 
speakers (see section 1 above) was very high. 



Suggested Improvements for Future Workshops 

By and large, the most frequent comment made here as well as 
in the informal discussions was that the workshop be reduced from 
two weeks to one week. The participants were rather emphatic that 
this not be done at the expense of the substantive material presented. 
They felt, however, that by lengthening the workshop day and even 
possibly having one or two evening sessions the basic material could 
be presented in one week's time. Some other suggestions were: begin 
the s/ jsion with a general overview, have participants receive materials 
prior to the beginning of the workshop (we would have liked to do this 
but were unable to do so because it was not until the last minute that we 
knew funding was available), provide more consultant help to work on 
individual state plans, have a three -day follow-up workshop in the 
summer of 1972 to discuss plans and problems as well as to share in- 
formation, spend more time in the technical areas as well as in the 
affective domain, and have each state briefly describe what it is doing 
at the beginning of the workshop. 

Many of these suggestions are good and should be incorporated for 
future workshops. For example, if each state was to describe briefly 
on the opening day what they are doing, it would allow other participants 
to meet and discuss similar problems and concerns. .And, by having 
the materials available before the workshop begins, the participants 
could begin to "zero in u on their particular problems. We concur heart- 
ily with many of these recommendations. 

Most Valuable Experience 

As would be expected with a diversity of persons attending the work- 
shop, those experiences valuable to some would be less valuable to 
others. Nevertheless, the participants were very much in agreement 
that the most valuable experience was the opportunity to hear the various 
speakers. From them, they felt that a great deal of knowledge was re- 
ceived. Regarding specifics, the participants said that the material 
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presented on criterion versus norm-referenced tests, on sampling, 
and on data analysis were very valuable. They also felt that the 
opportunity to interact with the staff and other personnel was valuable. 

The least valuable experiences were exercise writing, hearing 
the New York plan, and working on the individual state plans. These, 
however, were less than 20% of the participants 1 feelings and resulted 
in some instances from the fact that some states had only one repre- 
sentative who was not involved in preparing exercises or who did not 
have direct responsibility for conducting his state's assessment. 

Future Workshops 

All participants stated that they would recommend this type of 
workshop for other State Department personnel. They also felt that 
workshops of this nature should be conducted in the future. 
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Conclusions and Recommendations 



On the basis of the workshop evaluation as well as from informal 
discussion with the participants, it is evident that the workshop was 
very successful. This does not mean that there were no problems 
there were some minor ones which can be easily corrected. The par- 
ticipants were very pleased with the overall structure, the content, 
and the speakers. They felt that they obtained a better understanding 
of National Assessmer 1 in general,- and their local assessment plans 
in particular. This, of course, was the intent of the workshop. It 
was not the directors' intent to proselytize the participants for a par- 
ticular plan. Rather, it was our intent to give the participants a gen- 
eral view of assessment so that they would be better able to implement 
the information presented at the workshop insofar as their state plans 
were concerned. In this way, hopefully, future state and local assess- 
ments would be more valid and reliable and would avoid some of the 
problems experienced by others. As many of the participants said 
"there is no reason to re-invent the wheel". 

Our recommendations are three in number: 

1. Future workshops should be conducted using a plan very similar 
to the one just completed insofar as structure and content is 
concerned. More and more states as well as local school 
systems are involved in some form of assessment and it would 
be unfortunate indeed if they could not capitalize on what has 
already been learned in National Assessment as well as in other 
state assessment plans. We also recommend that the next 
workshop be condensed into one week by lengthening the work- 
shop day. 

2. The participants were very enthusiastic about the idea of a 
brief follow-up workshop next summer where they personally 
could get together and discuss some of the plans already 
underway, those in the planning stage, the problems encoun- 
tered, the solutions found, etc. If this were possible, we 
would recommend an overlapping session between the "new" 
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and "old" workshops so that the new personnel would be 
able to profit from those who have already begun to imple- 
ment state assessments. 

We recommend that, if possible, states should send a team of 
two persons --an evaluator /planner and a curriculum special 
ist to future workshops. Those states that had only one rep- 
resentative felt that they did not have enough knowledge to 
be able to derive maximum benefit from the workshop. Some 
of the material presented is directed at the evaluator while 
other material is directed particularly to the curriculum 
specialist (objectives, exercise development). 
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Colorado Assessment Program: 
A Five -Year Plan 



Long-range objectives of the assessment may include the following: 

1. To identify educational needs with such precision and 
relevance as desired for, (a) allocation of resources 
to remedy the needs, and (b) reassessment over a 
period of time to determine effectiveness of remedial 
efforts. 



2. To provide the State Board of Education, Legislature 
and interested others with answers to their questions 
regarding educational quality, needs, and achievements 
in Colorado, so as to increase support and understand- 
ing. 

3. To develop capabilities among district personnel in 
processes and procedures of assessments leading 
toward local assessment systems operational in each 
district in Colorado. 

4. To implement the Colorado Accountability Act, enacted 
in June 1971, establishing programs of educational ac- 
countability at the state and district levels of responsi- 
bility for defining and measuring educational quality, 
and reporting results to interested persons and agencies. 
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A Model for Planning a State Assessment Program 
for the State of Idaho 



Intr oduction ; 

At this point in time, Idaho has no formalized plan for state 
assessment nor does it have any definite plans that we are aware 
of to proceed in this direction in the immediate future. Our as- 
signment for this workshop was to become knowledgeable about 
National Assessment and other state plans and to report this to 
the Idaho State Department of Education. Therefore, our report 
deals with a model for planning that might be utilized in arriving 
at this decision to start a state assessment program. Also, we 
hope to consider some of the major issues that would have to be 
taken into consideration. 

1. Present information collected at the ECS Assessment 
Workshop to the coordinating council of the Idaho SDE. 

Based upon the decision of this group, the plan could 
terminate at this point or proceed as follows: 

2. The Coordinating Council or the CSSO will assign SDE 
personnel or units the responsibility of coordinating 
the development of a model for planning a State As- 
sessment Program. 

2. 1. Involvement of SDE personnel which would 
be orientation to assessment, estimation 
of time commitment needed for assistance 
in model development, etc. 

2.2. Review literature on National, State, and 
Local assessment programs. 

2. 3. Compile budget for developing a model for 

planning an Idaho State Assessment Program. 

2 # 4. Establish criteria for number and representa- 
tion of members for a State Assessment Plan- 
ning Task Force. 

2.4. 1. Task Force to be appointed by State 
Board of Education 

2. 5. Compile data bank of resources for Task 
Force to draw from during their work. 
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Idaho cont'd 



2. 5. 1. 



Consultants from areas of assessment. 



2. 5. 2. Title III, ESEA Regional and State Needs 



2. 5. 4. LEA Needs Assessment Package. 

2. 5. 5. Accountability Pilot Project. 

2. 5. 6. LEA Self -Analysis Pilot Projects. 

2.6. Develop strategies for the order in which the is- 
sues will be considered by the Task Force and a 
tentative timeline with completion date. 

3. Issues that require Task Force consideration. 

3. 1. Develop State Assessment Program vs. con- 



tract with Student Information System (SIS) 
utilizing available test data in the LEA's. 

Based upon their decision, the Task Force work 
could terminate at this point upon the completion 
of specifications for the SIS contract. 



3.2. Funding (allocation of specific amount vs. 
quality program). 

3.3. Determine target population (who will be 
assessed), 

3.4. Determine whether assessees will be classified 
by grade or age. 

3.5. Categories to be included in assessment, pos- 
sibly prioritized. 

3.6. Norm-referenced vs. criterion-referenced vs. 



3.7. How data are to be used? 

3. 8. Concept of adopt, adapt, (from other programs), 
or create. 

3.9: Using in-hour>e capability vs. contracting. 



Assessment. 



2. 5.3. 



LEA Planning Program sponsored by 



SDE. 



combination. 
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3. 9. 1. Developing objectives. 

3. 9. 2. Exercise development. 

3. 9. 3. Sampling. 

3. 9.4. Administration of materials. 

3. 9. 5. Scoring. 

3. 9. 6. Analysis of data. 

3. 9. 7. Reporting. 

3. 9. 8. Data interpretation. 

3. 10. Based upon decisions reached in 3. 9. above, 
criteria and strategies must be developed for 
each item listed. 

Present model outlined 2-3 to SDE for go/no-go 
decision. 

SDE appoints Task Force. 

Task Force develops model for State Assessment. 
Model presented to SDE. 

Model submitted to State Legislature for funding. 
Implementation for model for state assessment by 



SDE. 
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OUTLINE OF IOWA ASSESSMENT 



I . General Overview 

The Iowa Assessment is designed to sample knowledges, skills, 
understandings, and attitudes of 4th, 7th, and 12th graders in science, 
the language arts, music, social studies, mathematics, and vocational 
and career education. In addition attitude toward school and self-concept 
will be sampled in the affective area. Activities for sampling in the 
psychomotor area are now being studied for future inclusion in the Iowa 
assessment . 

The cognitive or subject matter assessment will use items as they 
have been released, are now being released, or will be released from 
the National Assessment of Educational Progress (NAEP) . Instruments 
utilized in the affective assessment have been developed and are now being 
refined by the Instructional Objectives Exchange (IOX) . 



II. Assessment Administration 

The administration of the instruments will be by taping all instructions, 
and taping all stimuli and responses in the cognitive areas only to 
allow for confidentiality of responses as pupils silently read items in 
the affective area. Pupils will respond anonymously. The items will be 
administered to groups of 12 pupils outside the pupils' classrooms and 
in the absence of classroom teachers and building administrators. 

Specially trained exercise proctors will administer all aspects 
of the assessment. They will receive approximately 4 hours of in-service 
training by Department personnel in charge of the assessment. The 
proctors will come from the ranks of substitute teachers in areas 
relatively close to assessment sites. 



III. Sample Design 

The universe for the sample will consist of all full-time pupils 
enrolled in Iowa public schools in grades 4, 7, and 12 excluding 
special education and private school students (including those attend- 
ing public schools on a shared-time basis). A stratified, multistage, 
cluster sample of pupils will be selected and stratified by race, sex, 
geographic region, district size, and educational level of parent. In 
designing the sample three zones will be delineated: an urban zone, 
a suburban zone, and the remainder of the state (other) . A sample of 
schools will be selected independently from each zone and geographic 
area, some zones and each geographic area, constituting a separate 
stratum. Thus, for sampling purposes, thirteen strata will be defined. 



29A 




Iowa (cont'd) 



III . (continued) 

Within each stratum, the sample will be selected in stages. 
The number of stages will depend upon the stratum and the amount 
of information available. At the most, four stages will be used. 
First, school districts will be selected within stratum (except in 
the seven largest districts£which were specifically inserted in the 
sample so as not to exclude black!^ where each district constituted a 
stratum). This will be done by ordering the district by size in terms 
of total 1970-71 enrollment figures and selecting a sample in a syste- 
matic manner, thus assuring a representation of districts of different 
sizes. Within a sample school district, a particular building or 
attendance center will be selected. Within the sample building or 
attendance center, a particular section, class, or homeroom will be 
drawn. Finally, within the section, individual students will be 
selected. At each stage selection will be made with probability 
proportional to size in terms of enrollment or estimated enrollment. 
The selections of pupils within class will be made at random with equal 
probability . 

The proctors indicated above will be provided with lists of the 
names of pupils comprising each sample group. Substitute names will 
also be provided to help reduce attrition rates where pupils might be 
absent or otherwise unable to participate on the scheduled testing 
date. Generally two male and two female names will be provided with 
instructions to substitute within sex. When substitutes become 
exhausted within a given sex no further substitution will be made. 



IV. Rationale for Iowa Assessment 

Once statewide goals are established for education in Iowa the 
Department of Public Instruction (DPI) will focus on providing evalua- 
tion leadership and stimulating local education agencies (LEA's) to 
conduct needs assessments of their own. The ultimate goal of Iowa 
Assessment is to direct educational leadership and energy toward 
increasing the quality of education for boys and girls at the local 
school district level. A systems model which graphically portrays the 
logical flow of tasks and activities has been developed by the Iowa 
DPI and is in trial use by one school district at this time. 




The Development of MAPS, Maine Assessment Tu 



Planning For Schools: Summary of Comments 



A Brief Sketch Of Where We Have Been , 

MAPS, Maine Assessment In Planning For Schools is based on 
the following: 

1. Responsibility for education in the United States has been 
reserved to the individual states. Traditionally, Maine has been 
concerned not only with establishing minimum standards and controlling 
subsidies. It has also concerned itself with the improvement of the 
quality of education. 

2. There is growing and serious concern regarding man's ability to 
understand himself in relation to the rapid social and technological 
changes of the world. The individual's failure to continually 
reeducate himself will result in his becoming a disfunctional member 
of society. Consequently, the extent to which schools renew their 
educational programs will have a profound effect on the ability of 
society to cope with the technological and social changes presently 
taking place. 

3. Gaining a total perspective of the educational program, a complex 
and endless task, is necessary. In reality, neither improvement nor 
new direction can be achieved without undertaking this task. 

4. Assessment, as part of the planning process, is human resource 
oriented. Planning with rather than for is the emphasis. 

The planning and evaluation proposal design includes a director, 
planner, evaluator and systems analyst. In addition it provides for 
the formation of a Planning Team which is representatives of the State 
Department of Education. One task of the Planning Team is to assess 
the most urgent educational needs of Maine. These needs stem from the 
difference existing between desired goals and current conditions. And 
the success of assessment effort, as part of the planning process, can 
only be measured in terms of the progress that is made in achieving 
defined educational objectives. 

To facilitate the development of MAPS the Planning Team has 
addressed itself to five board areas of study: 



1. curriculum 

2. personnel 



3. demography 

4. finance 



5. supportive 
services and 
facilities 
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Maine (cont'd) 



Then task force groups have been formed which include members of 
the Planning Team and extending the involvement to others in the 
Department who have expertise or interest in a particular area. The 
five task forces have adopted a common format on which to proceed: 

1. defining the problem 

2. stating objectives 

3. formulating activities and/or studies. 

A major area of concern of the task forces is the lack of output 
data, in terms of student achievement and educational objectives. 



B - A Brief Sketch of Wha t We Muat Do and Some Alternatives 

1. MAPS should derive output data, either norm referenced or criterion 
referenced, relative to defined objectives. Matrix sampling techniques 
would generate, at relatively low cost, usable data for decision making. 
Some population components should be more heavily sampled. For example, 
disadvantaged groups, Indian's and bi-lingual groups. A modular assess- 
ment package should be designed that could include the following broad 
subject areas: reading, mathematics, social studies, citizenship, 
literature, music, art, career and occupational development and writing. 
Serious consideration should be given to assessment not only in the 
cognative areas which are relatively easy to measure but to measurement 
in the affective and psychomotor domains which are far more difficult 

to measure. 

2. Specific objectives should be developed relative to areas to be 
assessed. The objectives must be satisfactory to area specialists, most 
schools and thoughtful lay adults. 

3. An out-of-house advisory committee, truly representative of the 
Maine community, whose functions would include the development of 
assessment efforts, must be formulated. 

A. ^ There must be continued expansion of technical assistance to 
LEA's in planning and evaluation. Through these activities a high 
level of co-operation can be achieved. 

5. The team approach must be used to bridge the gaps between plann- 
ing, development of assessment devices, data gathering, data analysis 
and reporting. The reporting phase, in a sense the end product, is 
highly critical to success of an assessment and in providing direction 
for planning, development of assessment devices, data gathering and 
data analysis. 
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Maine (cont'd) 



NAPS is an exciting and potentially productive effort in defining the 
critical needs of Maine education for these reasons: 

1. it focuses on a board base of human resources on educational 
problems. . 

2. it provides the mechanism for evolving data about our educational 
system and immediately using it to improve schools. 

3. through increasing the awareness of the interrelationship of 
educational problems and concerns, it should facilitate cooperative 
efforts in planning programs to meet these needs. 
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Outline of a Plan to Assess Student 
Performance in Maryland 



The purpose of this outline is to list the general steps which 
need to be taken by the Division of Instruction to develop and con- 
duct the first phases of a state-wide assessment of pupil perfor- 
mance in selected subject disciplines. This outline is limited to 
functions within the Division of Instruction (DI) and should be con- 
sidered part of a larger assessment model which is currently under 
development by an inter -departmental task force. 

1. Goals 

The DI shall utilize goals already established by the 
Department. 

2. Objectives 

By July 1972, curriculum consultants and specialists 
shall have prepared content objectives for the various 
disciplines at specified grade levels. 

2. 1 Prepare a comprehensive list (50 to 150) of 
content (behavioral) objectives for each dis- 
cipline. Potential sources include: local 
and national guides and reports, teacher re- 
ports and comments, student reports and 
comments, reports by professional associa- 
tions, comments by parents and other indivi- 
duals. 

2.2 Submit the objectives lists to professionals, 
students and parents for rating. 

2. 3 Determine the relative importance of the ob- 
jectives as rated in 2.2 and compile final 
results. 

3. Exercise Development 

By July 1973 curriculum consultants and specialists 
shall have developed exercises which will be used to 
assess student performance on the various content 
objectives. 
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3. 1 Prepare prototype exercises in each discipline 

by adapting or modifying items already developed. 
Potential sources include: National Assessment, 
other states, University item pools. 

3.2 Evaluate prototype exercises through field tests. 
Finalize exercises in the various disciplines as 
a result of field tests. 

4. Sampling Design 

By July 1973 the Division of Research, Evaluation and 
Information Services shall have developed a research 
design for sampling student performance on the various 
exercises. 

4. 1 Stratify the school population within the State. 

Possible categories: Type of Community, size 
of community, socio-economic status, race, 
sex, age. 

4.2 Choose a sample of schools and specify a number 
of pupils in each school to be tested. 

5. Exercise Application 

By January 1974 the Division of Research shall have 
applied the content exercises in selected disciplines to 
the sample population. 

5. 1 Package exercises into sets which maybe 

applied to the respective age groups. 

5. 2 Train personnel who will administer packages. 

5.3 Apply exercises to sample population. 

6. Analysis and Reporting of Data 

By October 1974 the Division of Research shall have 
analyzed the data and prepared reports of the findings. 

6. 1 Collect data and prepare for processing. 

6.2 Analyze data and prepare for reporting. 

6.3 Prepare written report of the findings. 
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A Model for State Assessment of 
Education in Massachusetts 



The following represents a general outline of what could be 
a complete educational asse s sment cycle in Massachusetts. Each 
phase will demand a number of decisions by administrators within 
the department. A large number of these decisions will be dictated 
by the funds available to carry out this plan. 

Phase I. The Establishment of State Goa's for Education 

As an initial step toward a statewide assessment of educa- 
tion, the Department of Education will under take the task of 
updating the state's goals of education. This phase will en- 
tail three distinct activities. 

Activity One . 

The Commissioner of Education will form a task force of 
department personnel to review past goal statements and 
other reports and prepare a working document on goals. 

Activity Two . 

This working document will then be reviewed by at least 
three hundred leading citizens from a cross section of the 
state. Their recommendations will result in a Board of 
Education approved publication entitled, "Goals for Educa- 
tion in Massachusetts. 11 

Activity Three . 

The goals will then be distributed throughout the state. 

Phase II. The Development of State Objectives for Education 

The department, either by private contract or through 
department personnel, will develop a list of specific ob- 
jectives which will relate both to the goals of education 
for the state and to the various areas in the curriculum 
of a school. These objectives will be distributed widely 
throughout the state. To complete Phase II the following 
activities will be completed: 

Activity One . 

Establish a list of the various publics to whom these ob- 
jectives will be distributed. This list should represent 
at least thirty thousand people and should include either 



37A 




Massachusetts cont'd 



in total or by sample students, school administrators, 
teachers, legislators, school committee members and 
the general public. 

Activity Two . 

Construct a questionnaire or a series of questionnaires 
which will please each of the various groups listed above. 

Activity Three . 

Distribute and collect the questionnaires. 
Activity Four . 

Publish and disseminate the results of the questionnaire 
throughout the state. 

Phase HI. Planning 

The department, either by contract or through department 
staff, will develop a model by which it will assess the at- 
tainment of those objectives by students. The model will 
be developed after the department has decided what object- 
ives are to be assessed. Once determined the following 
activities should be completed in order to complete this 
phase. 

Activity One. 

The department, either by contract or through department 
staff, will develop criterion-referenced items for the chosen 
objectives. These items will be reviewed both by curricu- 
lum specialist and test specialist. 

Activity Two. 

The department, either by contract or through department 
personnel, will develop a sample which will render a satis- 
factory level of validity about the various elements upon 
which the department is reporting. 

Activity Three . 

The department, either by contract or through department 
staff, will organize the various test items into a test or 
series of tests not to exceed forty-five minutes in adminis- 
tration. Each student selected should be given questions in 
at least two areas. 
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Activity Four, 

Print test packages, design test and correcting procedures. 



Activity Five, 

Develop necessary public relations activities. 
Phase IV , Implementation 

The department, either by contract or through department 
staff, will execute the plan as outlined in Phase III by dis- 
tributing, monitoring, collecting and scoring the test. 

Phase V , Analysis 

The department, either by separate contract or through 
department personnel, will analyze the corrected test 
data. In completing this Phase the department will carry 
out the following activities. 

Activity One , 

Analyze each item stating a percent result for each group 
upon which we are reporting. 

Activity Two. 

Assemble curriculum specialists in each of the areas tested 
and draw from them opinion statements on each item and 
any general recommendations across all items or objectives. 

Phase VI. Reporting 

The department through staff personnel with the aid of 
private contractors will report the findings of the assess- 
ment study through the following activities. 

Activity One . 

Publish and distribute to all schools a technical journal 
listing the results on each test item. This report should 
include the percentage breakdown for all groups as well 
as all opinion statements from curriculum specialists. 

Activity Two . 

Publish and distribute a newsletter to all persons who 
participated in establishing the goals and objectives as 
well as other appropriate persons. This newsletter 
should outline in laymen's terms the results of the as- 
sessment. 
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Activity Three, 

Write and publish articles about the assessment pro- 
cedures and results. 

Activity Four. 

Prepare presentations for various groups within the 
state. 

Conclusion: 

The preceeding discription of a needs assessment model 
represents a broad discription of the elements within an assess- 
ment program. Under each phase the state has a variety of op- 
tions which could substantially effect the cost and effectiveness of 
this plan. 
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Statewide Needs Assessment - 



Texas 



Developed around planning procedures that permit flexibility 

A. At the present there are limited resources for assess- 
ment. 

B. Resources preclude the continuous assessment exemp- 
lified by national assessment. 

Assessment is goal oriented 

A. Operational goals 

1. Timely and reliable information for Legislature, 
State Board of Education, other decision-makers 

2. Establishing a systematic procedure for identify- 
ing educational needs and assessing the status of 
pupils in the identified areas. 

3. Encouraging the alleviation of educational needs 
by actions of school districts through the imple- 
mentation of a design for school improvement. 

B. Assessment centers on the Goals for Public School 
Education adopted by the State Board. 

Assessment as a part of planning procedures 

A. Procedures are based on identifying focal concerns 
which are prioritized to have an operational approach. 

1. Certain steps have been planned to surface 
and establish priority areas of concern. 

2. Realistically most of the priority areas of 
concern come from edict. 

B. After a priority area of concern has been identified 
operations are started to gather information about it. 

1. A set of ''study 11 questions are developed 

about the concern and these questions serve 
as a basis for needs assessment. 
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III. B cont'd 

2. A search for existing information is initiated, 
is conducted, and adequacy of information 

is determined by the extent it answers the 
study questions. 

3. If information gaps are left by the search of 
existing information, an information search 
(assessment study) is then designed and ini- 
tiated. 

4. The additional information gathered is analyzed, 
and an information document concerning the 
study questions is produced. 

5. Procedures to formulate conclusions are then 
implemented. 

6. The steps are not necessarily sequential for 
recycling may occur at any step. 

C. The eight year design - To establish a continuous and 
comprehensive assessment design, a flexible frame- 
work has been designed that suggests assessment stud- 
ies but is flexible enough to permit changes in directions 
and new situations that might arise. 

D. Dimensions considered in the eight year design 

1. Time to provide a long range framework for 
making assessment decisions. 

2. Target populations - to insure that various 
subgroups are considered. 

3. Areas of concern - the activities presented 
in the design are qualified only in the number 
of the areas of concern to be dealt with. 

IV. An example of an assessment in an area of concern 

A. The status of pupils in regard to reading and mathe- 
matics has been established as an area of concern 
because no substantive statewide data exist. 

B. The following tentative decisions have been made 
about the assessment of reading and mathematics. 
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1. Assess sixth grade in the public schools and 
draw a sample based on ethnicity, geographic 
location, size of community, and affluence as 
determined by title eligibility. The sample 
should provide both statewide and regional in- 
formation. The sample will be drawn to in- 
clude school campuses that include sixth grade- 
ers rather than on a random selection of sixth 
graders. 

2. Use criterion-referenced instruments that have 
been developed from objectives that have been 
determined as being valid for sixth grade pupils 
in Texas schools. 

3. To entice schools to participate, furnish an 
information feedback to schools about the 
individual pupils who participated from their 
campus. The feedback will consist of: 

a. school results by objectives 

b. diagnosis of individual pupils from 
an item analysis 

c. group and individual pupil prescrip- 
tions that are founded on the schools 1 
instructional materials 

V. Suggestions for the reading and mathematics assessment that 
have been derived from information covered in the state as- 
sessment workshop. 

A. To establish precise objectives as to what information 
needs to be gathered to answer study questions. 

B. To determine the sample size from these objectives 
and to ascertain that sample populations are sufficient 
enough to have an acceptable level of confidence. Also, 
to consider the "cluster effect. " 

C. To design a data analysis plan that will generate in- 
formation about the objectives. Also, plan to use 
"gronking' 1 in reporting some of the results. 
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D. To work with regional education service center guidance 
coordinators to establish as much preciseness in instru- 
ment administration as possible. 

E. To plan for a phase type of data reporting and to gear the 
reporting to various audiences. 

F . To use the panel procedure in interpretation of results. 
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West Virginia Assessment Model 

Phase I: Development of Present Assessment 

In 1967, the Governor of West Virginia appointed a task force 
to study alternative means leading to comprehensive education in the 
State. As an outgrowth of this study, and the ensuing recommenda- 
tions, the CSSO in 1 968 appointed a committee to review the exist- 
ing goal statements for education. The review by consultants, De- 
partment personnel, educational groups, and laymen culminated in 
a refinement of previous goal statements on the basis of shared ed- 
ucational philosophy. The goals as stated by the committee were 
approved and are known as "Goals for Education in West Virginia. " 

The goals, twenty-five in number, were translated into 41 
measurable objectives with the assistance of West Virginia Univer- 
sity staff. Each goal was measurable by means of at least one ob- 
jective. Each objective in turn was measurable through a set of 
criterion variables. Through the criterion variables, a point of 
reference was established: "where we were" at a given time. 
Therefore, educational need relative to a given objective was 
determined by the difference between "where we were" and "where 
we wanted to be. " This model for determination of needs is com- 
monly termed a discrepancy model. 

The Goals for Education in West Virginia dealt solely with 
the schools as an institution, not with changes in the pupil. To 
overcome this deficiency, a second set of goals was established 
jointly by the West Virginia Education Association and the De- 
partment. These goals were entitled "West Virginia's Goals for 
a New Era" , and dealt with pupil outcomes. 

The West Virginia Legislature, as an incentive to LEA 's 
to develop comprehensive educational programs, provides money 
outside the school aid formula. To obtain this money, the LEA's 
describe the status of their schools relative to minimum require- 
ments of the Comprehensive Education Program (CEP). This 
program establishes minimum outcomes (objectives) in each of 
19 instructional areas. These pupil-oriented objectives were 
derived from "West Virginia's Goals for a New Era" and were 
designed to be a generation source of data for outcome assess- 
ment. 

Phase II: Addition of Pupil Need Assessment 

By using the present CEP program objectives and the State-" 
County Testing Program, the Department will assess pupil needs 
in 1971-2. At present, battery-type achievement tests are given 



45A 



West Virginia cont'd 



to EVERY student enrolled in EVERY school in grades 3, 6, 9, and 
11 each year. The remaining part of this paper is devoted to a des- 
cription of an operational procedure that will result in a pupil needs 
assessment based upon data available through the established State 
testing program. 

An approach to pupil-needs assessment: Cognitive domain. 

1. The CEP program objectives for selected instructional 
areas will be assessed using selected items from the 
annual State-County Testing Program. 



2. The State-County Testing Program will be administered 
to each pupil in selected grades statewide. 

3. The Executive Council of the Department will select 
CEP instructional areas for assessment. 

4. The Executive Council will appoint an advisory council 
whose membership will include instructional specialists, 
university personnel, consultants, and laymen. The 
advisory council will select CEP objectives to be mea- 
sured. Then, they will select test items from the achieve- 
ment tests that will stand as objective criterion variables. 

5. The scoring of separate answer sheets for each pupil will 
be a function of the Division of Data Processing. 

5. 1 An optical scan method will be used. 

5.2 The total response for the grade grouping will 
be given for each item selected by the advisory 
council. 

5.3 The p-values for each selected item will be 
computed. 

6. The item response counts and p-values will be dis- 
seminated. 

6.1 A state-item response count. 

6. 2 Seven regional item response-counts. 

6.3 An item response count reported to individual 
LEA's. Only the LEA may release the data. 
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Phase III: Future Analysis and Reporting 

A Goal Attainment Index will be used to report assessed 
pupil needs for each LEA for each objective (Pennsylvania 
model). 



L« The formula for computing the expected percentile 
of correct respondents to items representing objec- 
tive criteria variables will be developed with the 
assistance of outside consultants. 



1. 1 Community variables that will be used in 
predicting percentile of correct respond- 
ents will be determined. 

1. 2 Procedure for analysis of data will be 
determined. 



2. A graphical representation of the data for each LEA 
will be prepared. 

2. 1 For each objective the expected percentile, 

the actual response percentile, and the State 
mean will be shown. 

2.2 The difference between the ''expected' 1 and 
"actual" percentiles will represent the as- 
sessed need. 

3. The significance of the assessment will be discussed 
with members of each LEA's staff by a Department 
consultant. 

3. 1 Upon request, Department, university, and 

outside consultants will assist the LEA in 
making inferences. 

3.2 More meaningful curriculum decisions will 

be made on the basis of assessed pupil needs. 

Phase IV: Future Pupil Needs Assessment: Affective and Psycho- 
motor 
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Wis con sin-Oregon 
Assessment to Determine Accountability 



Before outlining decisions and/or procedures, it is necessary to inject 
my bias at this point. I feel the assessment must have value to the admin- 
istrators at the school level. I would want this data to reflect the quality 
of administrator's students; therefore, reflecting his utilization of resources 
and processes. To accomplish this goal, it is necessary to compare his out- 
puts to other school's outputs, while attempting to categorize schools by 
inputs. In summary, I want to work with data that is as objective as possible 
not subjective, i. e. , where quality performance is determined by actual com- 
parison not by opinions of "experts". 

Decisions and/or Procedures (no order established due to parallel decision 
points ) 

1. Develop support of State Superintendent and Department. 

2. Establish policy making body (Curriculum and Research Bureaus) . 

3. Establish advisory committee (composed of teachers, superinten- 
dents, students, public). 

4. Establish legal bases for operation (assessment is mandated or 
volunteer, support from legislators). 

5. What will be contracted and what will be done in house (items, 
objectives, data analysis, scoring, etc.). 

6. Compile a budget. 

7. What areas are we assessing? a) reading, mathematics (cog- 
nitive skills) b) self- concept, concept toward school (affective) 

8. Collect relevant literature on existing assessment and account- 
ability models and collect research relating input factors to output. 
(Coleman, Michigan's, etc. ) 

9. What grade levels or ages will be assessed (3rd and 6th to begin). 

10. What data are available at present in state and can be utilized. 
(Title III & I, demographic, no state-wide testing program) 

11. Establish objective bank (adopt, adapt, create; let superintendent 
and community determine objectives). 

12. Establish item banks to measure objectives (utilize all existing item 
banks; National Assessment, Pennsylvania, University of California 
Los Angeles, COMBAT, etc.). 

13. Establish sampling procedures (dependent on variables such as 
cost accuracy, etc. ). 
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14. Establish test administration procedures (teachers vs. contract- 
ing, procedures for public relations), 

15. Develop testing packages. 

16. Establish procedures for scoring (computers, teachers, etc.). 

17. Determine input factor in which to classify schools (STOE, 
cost/pupil, income, socio-economic). 

18. Determine data analysis procedure (programming, cost, infor- 
mation wanted, etc. ). 

19. Establish reporting and sissemination procedures (geared to 
principal or superintendent, similar to Pennsylvania, only no 
need for regressions). 

20. Develop follow-up procedures where help can be provided (cur- 
riculum, resource distribution, etc.). 

21. How often will assessment take place? 

22. Develop PERT and CPM charts. 



The state department should place in the hands of the principal and /or 
superintendent a document indicating the quality of education in his school 
as compared with the quality of education of similar schools. This could 
be in the form such as Pennsylvania, or as a deficiency rating from the 
best schools in his particular category. Once this document is in the hands 
of the principal or superintendent, it is his responsibility to investigate the 
better school's processes or consult the state department for help in im- 
proving the education of his school. It is hoped in this way that the results 
can be used in a more functional way. 
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ECS Assessment Workshop 



WORKSHOP EVALUATION 



The purpose of this evaluation is to determine the impact of your 
experiences here in relation to your local assessment needs. Because 
of a possible follow-up, we request that you identify yourself so that 
we can collate the results of the two evaluations* All information will 
be kept confidential. Only group results will be reported. Thank you 
in advance for your cooperation. 



1. Name of participant: 

2. State: 



a. Listed below are the nine major presentation sessions of this workshop, 
Please rank them in descending order of importance to you in the 
performance of your present and (possible) future work. 



Development of Objectives 
Colorado Assessment Plan 
Pennsylvania Assessment 
Administration and Scoring 
Reporting and Dissemination 



Exercise Development 
Sampling 

New York Assessment 
Data Analysis 



4. What are your general impressions of the value to you of the following: 



a. 


Overall workshop structure 


good 


average 


poor 


b. 


Workshop content 


good 


average 


poor 


c. 


Speakers/discussion leaders 


good 


average 


P* or 


d. 


Training methods 


good 


average 


poor 


e. 


Workshop materials 


good 


average 


poor 


f . 


Hearing other state 










plans (last 1-1/2 days) 


good 


average 


P oor 


6- 


Independent work sessions 


good 


average 


poor 


h. 


Living facilities 


good 


average 


poor 
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Comments : 



5. What improvements would you recommend for future workshops (content, 
speakers, training methods, facilities, organization, etc.)? 



6. What was of MOST value to you in this workshop? Why? 



7. What was of LEAST value to you in this workshop? Why? 



8. Would you recommend this workshop for other State Department 
personnel? Yes No 

Why /Why not?_ 



9. Pinal comments: 
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Deportment of Health, Educotion, ond Welfare 


Gront No. 

OEG-0-71-4554 


OATE OF THIS REPORTING PCRlOO 

FROM 6 /28 111 to 10/20/71 


NAME ANO AOORESS OF GRANTEE INSTITUTION 

Education Commission of the States 
1860 Lincoln St, Suite 300 
Denver, Colorado 80203 


TRANSACTION NO. 

71-0-E-3157 

INSTITUTIONAL 10 NO. 


PROJECT PERIOD 

from 6.28.71 to 7.23.71 


ED CHECK IF FINAL REPORT Part I 



1. Expenditures of DHEW Funds for this Report Period 



o. Personnel . 

(secretarial) 


* 232.00 


h. Alterations and renovations 




b. Consultont services 


2750.00 


'• 0,her see below 


821. 73 


c Equipment 








d. Supplies 255.00 
Pi»in«ng 17A r lR 


431. 15 


|. Totol direct costs 


6454. 76 


e. Travel, domestic 

(rnnfltiHanf^ 


2219.88 


k. Indirect costs: 

Rote x nSAWriTDC 




f. Trav el, foreign 




Rasa 1 




g. Patient care costs 




1. TOTAL 


$ 6454. 76 


2. Expenditures from Prior Periods (previously reported) 




3. Cumulotive Expenditures 


6454. 76 


4. Total Amount A worded - Cumulatively 


8020. 00 


5. Unexpended Balance (Item 4 less Item 3) 


156*. 24 


6. Unliquidoted Obligations 




7* Unobligated Bolance (Item 5 less Item 6) 




8.a* Cost Shoring Information - Grantee Contribution This Period 




b,X of Total Project Costs (Item 8a divided by total of Items 1 ond 8a) 




9.a. Interest/ Income (enclose check) 




b. Other Refundoblo Income (enclose check) 





Car rental 235. 13 

Communications 75. 00 



I hereby certify thot this reports true ond co.crttHc the best of my knowledge, ond thot oil expenditures reporlod herein hove been mode in 
occordonce with opproprloJoxfJVont policies ond for th)» purposes set forth in the oppllcotloo and oword documents. 
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ECS Assessment Workshop: Six-Month Evaluation 



Abstract 



A two-week assessment workshop was held in Boulder, Colorado 
on July 12-23, 1971 for personnel from state departments of education. 
Seventeen participants from ten states were in attendance. 

The National Assessment of Educational Progress was used as the 
primary assessment model. One day each was devoted to development 
of objectives, exercise development, sampling, administration and 
scoring, data analysis, and reporting and dissemination. In addition, 
three one-half day sessions each were devoted to presentation of three 
existing state assessment plans ~ Colorado, Pennsylvania, and New 
York. 

Conference participants prepared an outlined plan for assessment 
in their own states, utilizing whatever- model or variations seemed best 
to them. 

Approximately six months after the workshop, the participants 
were asked to complete a follow-up evaluation questionnaire. Thirteen 
participants from nine states returned completed questionnaires. All 
indicated that they prepared some type of report on -the simmer workshop 
which they presented to their superior. All but one indicated that 
changes, some attributable to the workshop, had taken place in their 
state assessment plans . In evaluating the various workshop presentation 
as was the case in the initial evaluation, there was a preference 
expressed for the more specific, technical presen La Lions. Many of the 
participants indicated the greatest impact of the workshop was that they 
obtained a clearer picture of the methodology involved in conducting a 
state or local assessment. The participants indicated the ways in which 
they perceived assistance could be given to state assessment by U.S.O.E. 
E.C.S., and N.A.E.P. All respondents indicated an interest in having a 
brief ^ follow-up workshop, the majority saying that they were quite 
certain they would be able to attend. 




Ackrtjv.'.lv.figj'.aritei 




the questionnaire. All were quite candid and many valuable ' 
suggestions were received for- future workshops. Some of the 
participants enclosed detailed information about the status of and 
progress m their individual assessment plans. 

The directors are grateful to each contributor for making 
this follow-up evaluation a success. ° 



Frank B. Womer 
Irvin J. Lehmann 
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Introduction 



The early 1970* s may wall be remembered in education as the 
"age of accountability." Assessment is a term commonly associated 
with accountability, a term that suggests a method for seeking evidence 
as to whether schools (or the total educational process) are being 
accountable, are doing what they say they are doing. Many assess- 
ment efforts are in the planning stages; only a few have been imple- 
mented and have produced results. National Assessment is easily the 
most visible assessment effort in the country, and, in the opinion of 
many, the most _ carefully developed one. In spite of considerable initial 
hostility, National Assessment now is accepted and approved by roost of 
the educational community. 

National Assessment is designed to provide national results, 
regional results, and size of community results. It is not designed to 
provide state or school district results. Since it was designed for effi- 
cient information-gathering in large units, it does have potential specific 
utility for states (and possibly a few very large cities). It lias general 
utility as a "model" assessment for almost any sized educational unit. 
The potential utility of National Assessment as a model is above-and-' 
beyond its major tlirust of gathering direct evidence on the achievements 
of young people; it is a side benefit that needs to be exploited. 

Already the educational research community is well aware of 
National Assessment and is highly approving. National Assessment is 
appearing as a major topic in test and measurement textbooks and is 
being taught in college classes. 

Some states have established state assessments and many others 
are planning assessments. Only a few have expressed no interest. 

The utility of National Assessment as a "field" model for state 
level assessments is apparent to many. The Education Commission of 
the States (ECS) has received many inquiries about National Assessment 
as a model, about National Assessment materials, etc. ECS held one 
short state assessment meeting a year and a half ago and over 20 states 
were represented. 

To make National Assessment more accessible to the states, and 
to present alternative assessment models to the s tates a two-week 
assessment workshop for state department personnel was developed by 
ECS, funded by NCERU, and organized by Frank B. Womer and Irvin J. 
Lehmann in Boulder, Colorado from July 12 - 23, 1971. 

In an effort to improve future workshops, an evaluation form was 
completed by each of the participants on the last day of the workshop 
and again approximately six months later. Because the directors felt 
that some of the major impact of the workshop experience might not 
evolve until the participants returned to their- jobs, they decided 
to conduct a follow-up. This report is concerned with the 6-iranth 
follow-up . 
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IXirirn; tins first week oi' Dw::;r..ber .1971, the suir^r workshop 
participants v/ere sent a questionnaire. In the letter accompanying 
the questionnaire, they were requested to be as candid as possible 
when responding to the questions. Our intent was to obtain informa- 
tion that would permit us to make an evaluation of the various 
presentations. We did not wish platitudes if they were not deserved. 
Only with candid responses could we really evaluate the impact of 
the various presentations on existing or planned assessment programs. 

Because of the small number of participants (17 from 10 states), the 
data are reported in nurr.eri.cal fashion rather than as percentages. The 
responses are based on replies received from 13 participants in nine % 
states. Although we could assume that two of the respondents from 
states that had two or more members reflected the opinions of fellow 
members, we chose not to do this. 




Evaluation 



Approximately 6 months after the workshop the participants were a:;kod to 
complete an evaluation f or/a (Appeixlix A) . The purpose of this eval- 
uation was two-fold: (1) to provide the workshop staff with some 
objective (in addition to subjective) evidence concerning the impact 
of the workshop experience on the participants, and (2) to provide the 
workshop staff and Office of Education with feedback that would or 
could be of value for future workshops of this nature. 

The major questions asked in the evaluation form were as 
follows: 

1. On returning to your job after the workshop, what information 
or recommendations about assessment did you make to your CSSO 
or others in your state department? 

2. Have any changes taken place in your state assessment since 
July? Were any of these changes attributable to the workshop? 

3. Recommended improvements for future workshops? 

4. In what ways could U.S.O.E., E.C.S. , and N.A.E.P. be of 
help to you in developing state assessment? 

5. Would you be interested in, and could you attend a 2-3 
day follow-up workshop in the summer of 1972? 



Type of Report Made 

All the workshop participants reported that they prepared either a 
written, oral, or combination written-oral report to their immediate 
supervisor about the summer workshop. Some of the participants reported 
directly to their superior .in the office while others reported directly 
to -their CSSO. One state (Maine) sponsored a 2-day workshop on National 
Assessment to follow-up on the material presented at the summer workshop. 

Some of the comments made were as follows: 

"I met with other members of the department and helped to plan the procedures 
to be followed.... Much had been pre-planned but my suggestions were 
followed in many ways." 

"I wrote a memorandum The memorandum has been followed by many face-to- 
face meetings with members of the Bureau." 

"Both an oral and written report was made to our chief of planning, research, 
and evaluation unit. In our cigency, reports are required to be nnde 
through channels, thus cumrnunication can suffer. 11 

"Discussed N.A.E.P. with Commissioner on two occassions . " 
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All but one of the states reported that some changes had taken 
place in their state or local assessment plans since the July workshop. 
Some of these changes were felt to be attributable directly to toe 
material presented and discussed at the workshop. The changes vary 
from one state to another as might be expected. Some, states are 
attempting to relate output measures to input resources. One state 
reported that as a consequence of the workshop, they plan to gather age 
and grade information although they originally did not anticipate doing 
this. Many of the participants indicated that their thinking about the 
writing of objectives and exercises, instrumentation, analysis, and 
dissemination became clearer after attending trie workshop. 

Comments such as the following were typical: 

"Certainly a much greater understanding of assessment techniques by more 
of our people." 

"The analysis and reporting will be based on techniques learned at the ECS 
assessment workshop." 

"Worked on committee to use National Assessment material to demonstrate 
the use of criterion-referenced testing technique." 

"We are re-examining our sampling procedures." 

Two- thirds of the states (N= 6) indicated that they felt the changes 
that had taken place in their assessment plans could be attributed to 
the summer workshop. Some of the comments were as follows: 

"The awareness it developed in me." 

"Ways of asking questions, specific objectives, and sampling were quite 
helpful." ^ 

"The workshop provided a starting point in which to categorize the 
assessment tasks . " 

"The workshop has contributed to my personal skills in evaluation and 
planning." 

"Workshop enabled me to provide input to the (my) project." 

"The concept of using criterion-referenced instruments was reinforced at 
last summer's workshop. The ideas about meaningful interpretation and 
reporting techniques to be applied to the assessment information that 
we are using is a direct result of the workshop." 

"Our plans to incorporate criterion-referenced measures , do longi- 
tudional studies, critically review the literature and state a^-css- 
ment procedures, and particularly the emphasis upon study of New York 
and Pennsylvania models may be directly attributable to the ECS Work- 



The necessity to use c^itierion-refervnced inervuroo which nwans 
individual a:j::inis:ratio.n in soi.ie cases has increased concern for ~ 
a F^Biplirig coaign. The p.'Vis^nt testing programs test 100 percent 
of the! student population in a given grade." 

Zveti the two states reporting "no" were positive in the sense that 
the workshop was of some assistance to their state's assessment plans ■ 
or the participant's growth in the area of assessment. 

On the basis of the replies contained in the questionnaire as well 
as from personal conversations with some of the participants, it is 
readily evident that some of the experiences provided at the workshop 
resulted in modifications and/or alterations of existing assessment- 
programs. In some instances, these changes could be attributed directly 
to material presented at the workshop. In other instances, the workshop 
acted as a reinrorcing agency. Regardless of how the workshop affected* 
the participant, it is quite clear that the material presented was 
beneficial to trie participants. If it only helped them elijninate 
preplexing problems or avoid costly mistakes, we feel that it served its 
purpose. 

Evaluation of Vforkshop Content 

On the last day of the workshop, the participants were asked to rank 
the various topics covered. On the follow-up questionnaire, the participants 
were asked to indicate what topics they felt should be included in future 
workshops of this kind, what topics could be excluded, and what suggestions 
they could offer to improve the material. The results of this question 
are shown in -the table below. 



Topic 



Include m future 
workshops 



Objectives 2.2 

Exercise development 12 

Sampling 12 

Administration and scoring 9 

Analysis 12 

Reporting and dissemination 12 

Pennsylvania assessment plan V, 

Colorado assessment plan 6 

New York assessment plan H 



L'xclude in future 
workshops 



0 
0 
0 



The other three partici- 
pants didn't want this 
topic excluded — but 
wanted less emphasis 



0 
0 
1 
2 



5 



03 



As v.Msthe case in the initial evaluation of the vuei.ous topics 

discussed in th« workshop, 1:1*- more technical aspects of the course 

developing objectives, exercise development, sampling, analysis, and 
reporting— v^re perceived by the part Lcipants as being, of more value. 
In the written ccercents provided, many of the respondent j indicated that 
they v/ou].d like to see major emphasis placed on the- delineation of 
objectives, writing exercises, and a more thorough discussion of sampling 
and analysis. The respondents felt that there should be less emphasis 
on the presentation of the three state plans—Colorado, New York, and 
Pennsylvania—where assessment is already in progress. The comments indi- 
cated tliat the information about the plans were of value but the 
participants would like to obtain more information on the techniques 
used in other states. 

It is not surprising that the more teclmical aspects of the workshop 
were perceived as being of maximum value to the participants. The" majority 
of the participants readily admit that they have not been trained in these 
areas but yet are responsible for understanding (and sometimes actually 
be ng involved in) concepts of sampling, analysis, and development of 
objectives. Because of this, they favor receiving as much information as 
possible in these technical areas. No doubt they feel that discussions 
about other state assessment models can be readily obtained and understood 
with little technical background needed as a prerequisite. 

Some respondents sent us rather detailed reports about plans .in, 
and progress of assessment plans in their states. Others indicated that 
they are just begiiming to get organized and have nothing to publish at 
this time. Still others reported that they are in the preparation sta^e of 
a report. 

It is evident from the few reports received that considerable time 
and effort has been devoted to drawing up detailed specifications for 
state assessment programs. It would appear that one of the important by- 
products of the summer workshop was a growing awareness in the participants 
of the complexity of conducting a state assessment. The presentation of 
the N.A.E.P. model gave the workshop participants an opportunity to see 
what already has been done, to see some of the problems that could and did 
arise, and enough background information so that they could avoid some of 
the problems in their plans. One participant reported, "Many of the pitfalls 
which were discussed at the workshop helped us in developing a state plan." 
Another said, "....after working in this area, a plethora of new questions 
has arisen." 

It would appear that the major distinction between criterion and norm- 
referenced measurement was a focal point in the participants' experience 
at the workshop. Many commented that they now had a better understanding 
between the -two especially as it related to item construction and item 
analysis. The following comment is typical of those received to the question 

Please feel free to discuss the impact of last summer's workshop in your- 
own way . " 
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ways of cfcvclopiii£ more p;ocis<j assessment: techniques 
and ideas to consider when doing assessment. So, the 
workshop presented a poLpourri of ideas, from which each 
state could pick and choose the ones that would fit the 
the assessment situation in the individual state. Of 
course, as always, in a workshop such as this, friend- 
ships were created with personnel from other states 
and much help and information was gathered through 
this. Also, the importance of having someone you 
know personally to call upon for help is important.' 1 

Assistance from other agencies 

We all recognize that any assessment program can derive much 
valuable assistance from other existing agencies such as U.S.O.E., E.C.S 
and N.A.E.P. However, we are not always certain of the parameters in 
which persons perceive the form that this assistance should take. 
Normally, we are prone to think that it is primarily in the form of 
financial support. And this may be the case. But, we wanted to furnish 
other interested agencies with infonnation that they might find 
profitable in their dealings with and understanding of state assessment 
personnel. Following are the comments .made by respondents relative 
to specific ways in which other agencies could be of assistance to them. 

(a) U.S.O.E. 

,f Continue the ETS report (the one which surveyed plans in and progress 
of state assessment)." 

"Act as clearinghouse on goal identification and assessment procedures. 1 ' 
"Have more definite ideas about state assessment and allocate funds." 
"Provide consultants." 
"Fund regional workshops." 

"Put me on their nailing list to receive copies of any materials on 
evaluation." 

"Coordinating agency—provide technical, administrative and financial 
assistance. " 

"Develop national data bonk of objectives and test items." 

(b) E.C.S. 

"Develop a series of statements in support of contemporary mdnapeir^nt 
practices . " 

"Provide consultant help." 
"Encourage assessment at LEA level." 
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"Provide leadership." 



"to a study of state legislation and its implementation (sic) on 
assessment. 11 

"Act as a clearinghouse about state assessment information." 

"PR on importance of state assessment." 

(c) N.A.E.P . 
"Release more items . " 

"Update objectives and items." 

"Keep people informed." 

"Provide consultant help." 

"Develop state hookups with national assessment." 

On the basis of the responses received, it would appear that the 
workshop participants had a somewhat clear-cut, delineated set of goals 
insofar as assistance to be received from U.S. O.K. , E.C.S., and N.A.E.P. 
are concerned. Basically, the participants felt that the major 
contribution that U.S.O.E. could make was in terms of financial assistnao 
The respondents perceived E.C.S.\<> role as a clearinghouse of information 
on state assessment and as the agency to be primarily responsible for 
providing the leadership and encouragement for conducting state and 
local assessment programs. N.A.E.P. ! s role, as might be expected, was 
primarily to provide consultant help to the individual states. 

Future workshops 

All 13 respondents indicated that they would be interested in 
attending a 3-day follow-up workshop where the major emphasis would be 
on a discussion of progress and problems encountered in their individual 
assessment programs. The follow-up for these persons would overlap 
a planned workshop for new people In order that there could be a sharing 
of experiences and information that would be beneficial to all.- Eight 
participants indicated that they could attend, one was quite certain, 
and four stated that they would if they could obtain approval of their 
supervisor. 




Conclusions and R:ooinrneri'.latiori*j 



On the basis of the workshop evaluation as well a:; from informal 
discussion with the p.-irLicipants, it is evident that the workshop was 
very successful. This does not mean that there were no problems — 
there were some minor ones which can be easily corrected. The par- 
ticipants were very pleased with the overall structure, the content, 
and the speakers. They felt that they obtained a better understanding 
of National Assessment in general, and their local assessment plans 
in particular. This, of course, was the intent of the workshop, It 
was not 'the directors 1 intent to proselytize the participants for a par- 
ticular plan. Rather, it was our intent to give the participants a gen- 
eral view of assessment so that they would be better able to implement 
the information presented at the workshop insofar as their state plans 
were concerned. In this way, hopefully, future state and local assess- 
ments would be more valid and reliable rind would avoid some of the 
problems experienced by others. As many of the participants said 
ft there is no reason to re-invent the wheel, 11 

. All respondents indicated that they presented some sort of report 
on the workshop to their immediate supervisor (one actually discussed it 
with his CSSO). Many felt that subsequent changes in their state assessment 
plans were directly attributable to the information presented at the workshop. 
One might try to encapsulate the various comments regarding the impact 
of the workshop by saying that it gave the participants an opportunity 
to obtain a clearer picture of what a state assessment program involves. 

The technical portions of the workshop were perceived as being of 
most benefit. Although the participants felt that they learned some- 
thing from the three states reporting on their programs, most of the partici- 
pants felt that attention should be given to a di.scussion of the 
techniques being used rather than just a report of progress, per se. 

The participants also suggested ways in which the U.S.O.E., E.C.S., 
and N.A.E.P. could be of assistance to them in developing state assessment 
programs. U.S.0.E. f s role was perceived as primarily financial; E.C.S. f s 
as prijrarily public relations, and N.A.E.P's rale as primarily operational. 

Our recommendations are five in number: 

1. Future workshops should be conducted using a plan very similar 
to the one used in July, 1971 .insofar as structure and content- 
is concerned. The major emphasis should be on a discussion of the 
technical aspects of assessment— formulating objectives, exercise 
development, sampling, analysis, and dissemination. Less 
emphasis should be on administration and reporting. Possibly 
only one or two sta te plans already in exis tence should bo 
reported. 
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An attempt should be madts to structure future workshops so 
that they will be condensed into one week. 

A follow-up workshop was recommended by all the former parti- 
cipants to discuss plans ali^eady underway, those in the planning 
stage, problems encountered, and solutions found. 

E.C.S. and N.A.E.P. should attempt to coordinate and dissemi- 
nate information about state assessment programs. A moire 
intensive follow-up snould be made with the workshop participants 
to obtain information that will assist E.C.S. and N.A.E.P. in 
working with local states. One way of course, is to have more 
workshops of the kind conducted in 1971 and planned for the summer 
of 1972. 

Consideration should be given to the feasability of conducting 
short worksliops in individual states as a follow-up of the 
major summer workshop. Time commitments preclude the opportunity 
to discuss in any great detail specific concerns and problems. 



Follow-iy 
Ques tioru laire 



Frank B. Worncr 
Irvin J. Lelirnann 



December 1971 



1. When returning to your state last summer (or early fall) did 
do anything to present information or recommendations about 
assessment to your CSSO or others in your state department? 
Please explain. 



. Has any change taken place in assessment in your state since 
July? Please describe if yes; please explain if no. 
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If vou answered yer. to (2) wou2c ycu attribute any of that 
to last sunriHjr'j workshop? Please explain. 



c::-ir:r,e 



4. You have had about six months to reflect on the workshop. As 
you see it now, please answer the following three questions; 
assuming that another similar workshop may be arranged for 
states not represented last summer. 



Topics 


cuss ions) would 
you definitely 
want included 
in .such a work- 
shop? 


wild L Ld_LKb 

(discussions) 
would you 
leave out of 
such a work- 
shop? 


wnaL 

specific 
suggestions 
for i rnnTY^wp— 

ment do you 
have? 


objectives 








exercise 
development 








sampling 








administration 
and scoring 








analysis 








reporting and 
dissemination 








Pennsylvania 
assessment 








Colorado 
assessment 








Mew York 
assessment 
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Please feel free to discuss 



the iirpact of last suricer's work- 



shop in your own way. 



In what way(s) could the U.S. Office of Education or the E< 
tion Commission of the States or National Assessment be of 
help to you in developing state assessment? 

(a) USOE 



(b) ECS 



(c) NAEP 
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: Co.ild you attavl »: 
d A-:^;jL v.-Lth Surilriy, 

It v:ould bo i:i B.v-.:1'J2'-. 



Yes 



No 



Would be interested 
Could attend 
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